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Executive  Summary 


While  still  limited,  the  results  presented  in  this  report  provide  credible  quantitative  evidence 
that  Capability  Maturity  Model®  Integration  (CMMI®)-based  process  improvement  can  result 
in  better  project  performance  and  higher  quality  products.  The  results  are  drawn  from  a  total 
of  12  cases  from  11  separate  organizations.  These  include  5  with  home  offices  in  the  United 
States  and  6  located  in  Europe  and  Australia.  Their  process  improvement  efforts  cover  both 
small  and  large  organizational  units.* 1  The  organizations  do  business  in  a  variety  of  sectors 
and  domains  including  information  technology,  banking  and  financial  services,  automotive 
and  aerospace  engineering,  simulation,  and  training.  They  apply  CMMI  model  practices  to 
systems  integration,  systems  engineering,  and  software  development. 

Many  possible  measures  of  performance  can  be  used  to  demonstrate  the  impact  of  CMMI- 
based  process  improvement.  This  report  presents  results  that  organizations  have  shared  with 
the  Software  Engineering  Institute  (SEIsm)  or  with  the  wider  community  in  public  forums. 

We  have  categorized  the  results  into  four  primary  classes  of  benefits:  cost,  schedule,  quality, 
and  customer  satisfaction.  Evidence  about  return  on  investment  and  related  cost-benefit  mat¬ 
ters  constitutes  our  fifth  performance  category. 

The  following  list  summarizes  the  results  from  the  12  cases  for  each  of  the  5  classes  of  per¬ 
formance  measures. 

•  Cost:  Six  cases  provide  nine  examples  of  cost-related  benefits,  including  reductions  in 
the  cost  to  find  and  fix  a  defect,  and  overall  cost  savings. 

•  Schedule:  Eight  cases  provide  evidence  of  schedule-related  benefits,  including  decreased 
time  needed  to  complete  tasks  and  increased  predictability  in  meeting  schedules. 

•  Quality:  Five  cases  provide  evidence  of  measured  improvements  in  quality,  mostly  re¬ 
lated  to  reducing  defects  over  time  or  by  product  life  cycle. 


Capability  Maturity  Model  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 

1  Definitions  of  “organization”  vary,  so  care  must  be  taken  when  making  comparisons.  In  some  in¬ 
stances,  the  numbers  reported  may  refer  to  the  organizational  scope  of  an  improvement  effort.  In 
other  instances,  the  numbers  may  refer  to  a  larger  entity  of  which  that  effort  may  be  a  part. 

SM  SEI  is  a  service  mark  of  Carnegie  Mellon  University. 
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IX 


•  Customer  Satisfaction:  Three  cases  show  improvements  in  customer  satisfaction,  in¬ 
cluding  demonstration  of  customer  satisfaction  through  award  fees. 

•  Return  on  Investment:  Three  cases  report  positive  returns  on  investment  from  their 
CMMI-based  process  improvement. 
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Abstract 


There  is  a  widespread  demand  for  evidence  about  the  impact  and  benefits  of  process  im¬ 
provement  based  on  Capability  Maturity  Model1*  Integration  (CMMI®)  models.  Much  has 
been  documented  about  the  practice  of  CMM®-based  process  improvement  and  its  value  for 
the  development  and  maintenance  of  software  and  software  intensive  systems;  however,  the 
existing  information  is  sometimes  outdated  and  there  are  increasing  calls  for  evidence  di¬ 
rectly  based  on  CMMI  experience.  This  special  report  presents  selected  results  from  12  case 
studies  drawn  from  11  organizations.  While  still  limited,  the  case  studies  provide  credible 
evidence  that  CMMI-based  process  improvement  can  help  oiganizations  achieve  better  pro¬ 
ject  performance  and  produce  higher  quality  products.  The  report  also  describes  plans  for 
gathering  further  evidence  from  organizations  using  CMMI  models. 


®  Capability  Maturity  Model,  CMM,  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark 
Office  by  Carnegie  Mellon  University. 
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1  Introduction 


Serious  process  improvement  of  any  kind  requires  a  considerable  investment  of  time  and 
money  on  the  part  of  the  organizations  that  decide  to  pursue  it.  Capability  Maturity  Model® 
Integration  (CMMI®)-based  improvement  is  no  exception,  and  trustworthy  objective  evi¬ 
dence  about  its  benefits  is  essential  for  addressing  a  variety  of  concerns.  Increasing  numbers 
of  organizations  are  considering  using  CMMI  models  but  some  remain  skeptical  about  the 
value  of  any  model-based  process  improvement.  Others  are  uncertain  about  the  value  of 
adopting  the  CMMI  Product  Suite  over  its  source  models:  the  CMM®  for  Software  (SW- 
CMM)  Version  2.0  draft  C,  the  Electronic  Industries  Alliance  Interim  Standard  (EIA/IS)  731, 
and  the  CMM  for  Integrated  Process  Development  Version  0.98  [CMMI  Product  Team  02], 
[SEI 97],  [EIA  98],  [Bate  et  al.  97].  Still  others  need  evidence  to  build  commitment  and  to 
gamer  the  requisite  resources  within  their  own  organizations.  Furthermore,  a  growing  number 
of  organizations  use  such  information  as  part  of  their  ongoing  quantitative  management,  and 
as  input  for  improving  their  organizational  processes  and  technologies.  They  would  also  like 
to  be  able  to  compare  their  results  to  those  of  others. 

There  is  a  widespread  demand  for  credible,  objective  evidence  about  the  impact  and  benefits 
of  process  improvement  based  on  CMMI  models.  For  example,  when  asked  in  a  recent  ques¬ 
tionnaire  whether  their  organization  needs  return  on  investment  (ROI)  or  other  quantitative 
evidence  about  the  CMMI  models,  26  percent  of  the  554  of  those  who  answered  the  question 
said  that  they  “must  have  it,”  while  another  47  percent  said  that  “it  certainly  would  help  to 
have.”  Fourteen  percent  said  that  they  had  “already  built  a  good  business  case.”  Only  13  per¬ 
cent  said  that  “it’s  not  a  real  issue  for  us,”  typically  because  their  organizations  already  had 
decided  to  adopt  the  CMMI  Product  Suite  for  other  reasons  [Chrissis  et  al.  03]. 

We  already  know  a  good  deal  about  process  improvement  based  on  the  SW-CMM  and  the 
value  for  the  development  and  maintenance  of  software  and  software  intensive  systems.  Nu¬ 
merous  case  studies,  conducted  over  more  than  two  decades,  provide  evidence  of  sometimes 
very  substantial  improvements  in  product  quality  and  cost  effective  delivery  that  accompany 
corresponding  enhancements  in  process  discipline  [Benno  &  Frailey  95],  [Butler  95],  [Dion 
92],  [Dion  93],  [Herbsleb  et  al.  94],  [Humphrey  et  al.  91],  [Lebsanft  96],  [Lipke  &  Butler  92], 
[McGarry  et  al.  98],  [Wohlwend  &  Rosenbaum  93],  [Jung  &  Goldenson  03]. 


Capability  Maturity  Model,  CMM,  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark 
Office  by  Carnegie  Mellon  University. 
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A  growing  number  of  more  broadly  based  studies  systematically  make  similar  comparisons 
across  many  projects  or  organizations  [Clark  97],  [Deephouse  et  al.  95],  [El-Emam  &  Gold- 
enson  00],  [El-Emam  et  al.  01],  [Goldenson  et  al.  99],  [Goldenson  &  Herbsleb  95],  [Harter  et 
al.  00],  [Herbsleb  et  al.  97],  [Krasner  99],  [Krishnan  &  Kellner  99],  [Lawlis  et  al.  95].  These 
studies,  too,  find  evidence  of  considerable  differences  in  product  quality  and  efficient  deliv¬ 
ery  that  vary  predictably  with  differences  in  process  capability  and  organizational  maturity. 

That  said,  much  of  the  existing  evidence  is  rather  dated  and,  as  more  organizations  consider 
using  CMMI  models,  there  are  increasing  calls  for  evidence  based  on  direct  CMMI  experi¬ 
ence.  The  purpose  of  this  special  report  is  two-fold:  to  present  preliminary  results  about 
CMMI  impact  and  to  describe  ongoing  and  future  work  in  this  area. 

Section  2  provides  a  synopsis  of  the  case  studies  on  which  this  special  report  is  based.  Sec¬ 
tion  3  follows  with  a  brief  discussion  of  how  an  organization  can  demonstrate  the  impact  of 
CMMI-based  process  improvement  on  project  performance  and  product  quality.  Section  4 
presents  results  from  a  series  of  case  studies  that  are  publicly  available  and  from  others  being 
conducted  in  collaboration  with  the  Software  Engineering  Institute  (SEIsm).  Section  5  de¬ 
scribes  other  work  underway  at  the  SEI  and  elsewhere  to  provide  credible  evidence  about  the 
impact  of  process  improvement  based  on  CMMI  models.  Section  5  also  contains  a  brief  dis¬ 
cussion  of  the  limitations  of  the  current  work  and  plans  for  the  future. 


SM  SEI  is  a  service  mark  of  Camegie  Mellon  University. 
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2  This  Report 


Most  of  the  results  presented  in  this  document  are  drawn  from  twelve  case  studies.  All  but 
one  are  from  recent  papers  and  presentations.  Three  cases  come  from  an  ongoing  SEI  series 
of  collaborative  studies  that  were  available  at  the  time  of  this  writing. 


Most  of  the  case  studies  in  this  report  are  based  on  evidence  provided  by  organizations  that 
have  adopted  the  CMMI  Product  Suite.  We  have  also  included  four  recent  examples  based  on 
experience  with  the  SW-CMM.  The  SW-CMM  is  a  major  source  for  the  CMMI  models,  and 
much  of  the  same  content  is  present  in  CMMI  models.  Evidence  based  on  the  SW-CMM  may 
also  remain  compelling  to  those  who  still  are  skeptical  about  any  CMM-based  process  im¬ 
provement.  In  addition,  all  of  the  organizations  that  reported  benefits  from  the  use  of  the  SW- 
CMM  are  currently  making  the  transition  to  the  CMMI  Product  Suite. 


There  is  less  quantitative  evidence  about  CMMI  model  content  that  has  its  legacy  in  systems 
engineering,  but  we  have  done  our  best  to  include  such  evidence,  both  in  this  report  and  the 
larger  set  of  in-depth  case  studies  described  in  Section  5.  While  much  of  the  evidence  still 
remains  qualitative  and  anecdotal,  we  also  address  the  value  added  by  integrating  processes 
and  practices  across  disciplines  and  across  a  wider  organizational  scope. 
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3  Demonstrating  Impact  on  Performance 


Process  improvement  based  on  CMMI  models  may  be  demonstrated  in  various  ways.  Some 
organizations  have  established  new  processes  or  changed  existing  processes  as  a  result  of 
making  the  transition  to  the  CMMI  Product  Suite.  Others  have  broadened  the  organizational 
scope  of  their  improvement  efforts,  through  the  integration  of  systems,  software,  hardware, 
and  related  disciplines.  Process  changes  also  sometimes  predate  the  transition  to  CMMI 
models,  perhaps  especially  for  those  who  are  among  the  first  to  have  been  appraised  at  higher 
maturity  levels.  For  example,  some  organizations  had  already  established  processes  that  were 
not  emphasized  or  emphasized  differently  in  the  SW-CMM.2  In  one  sense,  their  experiences 
cannot  be  attributed  to  CMMI  adoption,  but  their  results  certainly  show  the  importance  of  the 
best  practices  that  are  articulated  in  CMMI  models. 

Analytic  approaches  also  differ.  Some  organizations  make  comparisons  over  time,  across  ma¬ 
turity  levels,  or  other  major  process  milestones.  Others  focus  on  selected  project  or  organiza¬ 
tional  processes  that  can  be  mapped  to  CMMI  model  practices.  In  addition,  qualitative  refer¬ 
ences  are  made  to  things  done  differently  as  a  result  of  the  transition  to  CMMI  models. 

Similarly,  many  possible  measures  of  performance  can  be  used  to  demonstrate  the  impact  of 
CMMI-based  process  improvement.  Organizations  use  the  measures  that  make  the  most 
sense  given  their  particular  business  goals,  and  organizations  often  describe  CMMI  benefits 
in  different  ways. 

Figure  1  is  a  high-level  depiction  of  the  impact  of  CMMI-based  process  improvement.  The 
costs  of  process  improvement  are  illustrated  in  the  upper  left.  Process  capability  and  organ¬ 
izational  maturity  are  depicted  in  the  center  box,  and  the  box  on  the  right  illustrates  the  bene¬ 
fits  of  process  improvement.  The  costs  and  benefits  also  can  be  combined  to  calculate  ROI  or 
related  measures,  as  shown  in  the  box  on  the  bottom  of  the  figure. 


2  In  addition  to  the  Engineering  process  areas,  these  notably  include  Decision  Analysis  and  Resolu¬ 
tion  (DAR),  Risk  Management,  the  maturity  level  4  and  5  process  areas,  the  inclusion  of  Meas¬ 
urement  and  Analysis  at  maturity  level  2,  and  the  treatment  of  several  of  the  generic  practices. 
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Figure  1:  High-Level  Model  of  CMMI  Impact 


We  have  classified  the  performance  measures  into  five  broad  categories  for  this  report.  As 
shown  in  the  detail  under  benefits  in  Figure  1,  potential  benefits  of  process  improvement 
might  accrue  with  respect  to  cost,  schedule,  quality,  and/or  customer  satisfaction.3  The  fifth 
category  is  return  on  investment  and  related  measures,  as  shown  in  the  bottom  box. 

Organizations  typically  seek  to  optimize  some  combination  of  the  four  primary  classes  of 
benefits,  each  of  which  can  be  refined  to  include  a  number  of  more  specific  measures.  For 
example,  one  organization  may  be  more  interested  in  reducing  the  costs  of  its  products  and 
services  while  another  may  be  concerned  about  having  more  predictable  project  costs  or 
schedules.  Other  refinements  of  the  four  basic  categories  might  include  shorter  cycle  times, 
reduced  defects,  increased  productivity,  reductions  in  rework  and  concomitant  effort,  en¬ 
hanced  functionality,  and  maintainability.4 

As  also  shown  in  the  detail  under  costs  in  Figure  1,  there  are  always  costs  associated  with 
implementing  and  institutionalizing  process  improvement.  While  cost  data  are  available  less 
often  than  evidence  of  benefits,  organizations  frequently  are  concerned  about  ROI  and  related 
cost-benefit  matters;  hence,  they  constitute  our  fifth  category  of  performance  measures. 


3  Relatively  few  organizations  measure  customer  satisfaction  systematically.  Neither  does  it  fit  com¬ 
fortably  into  any  single  one  of  the  other  three  high-level  benefit  categories;  however,  it  is  impor¬ 
tant  enough  in  its  own  right  that  we  call  it  out  here  as  a  primary  class  of  the  potential  benefits  of 
process  improvement. 

4  Other  possible  performance  measures  such  as  employee  morale  or  process  compliance  per  se  do 
not  map  well  to  the  simple  benefits  classification  presented  in  Figure  1;  however,  we  do  discuss 
them  in  the  case  vignettes  in  Section  4. 
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4  Preliminary  Results 


The  results  that  follow  are  presented  in  two  ways.  In  Section  4. 1  we  present  a  tabular  sum¬ 
mary  organized  by  our  five  high-level  classes  of  performance  measures,  namely  product  cost, 
project  schedule,  quality,  customer  satisfaction,  and  return  on  investment.  These  are  followed 
by  a  series  of  descriptions  that  provide  more  detail  about  each  organization.5  While  detailed 
quantitative  evidence  is  not  always  provided  publicly,  each  of  the  organizations  included  here 
makes  reference  to  the  existence  of  such  evidence  in  the  presentation  of  its  results. 

4.1  Overview  of  Results 

4.1.1  Cost 

This  category  covers  instances  where  organizations  report  reductions  in  the  cost  of  final  or 
intermediate  work  products,  reductions  in  the  cost  of  the  processes  employed  to  produce  the 
products,  and  general  savings  attributed  to  model-based  process  improvement.  It  also  in¬ 
cludes  increased  predictability  of  costs  incurred.  More  detail  about  each  of  these  instances 
can  be  found  in  the  organizational  descriptions  in  the  next  section. 


Table  1:  Summary  Benefits  and  Impact:  Cost 


Result 

Model 

33%  decrease  in  the  average  cost  to  fix  a  defect  (Boeing,  Australia) 

CMMI 

20%  reduction  in  unit  software  costs  (Lockheed  Martin  M&DS) 

CMMI 

15%  decrease  in  defect  find  and  fix  costs  (Lockheed  Martin  M&DS) 

CMMI 

4.5%  decline  in  overhead  rate  (Lockheed  Martin  M&DS) 

CMMI 

Improved  and  stabilized  Cost  Performance  Index  (Northrop  Grumman  IT1) 

CMMI 

Saved  $2  million  in  first  6  months  after  reaching  CMM  ML3  (Sanchez  Computer  Associates,  Inc.) 

SW-CMM 

20%  reduction  in  average  cost  variance  (Thales  Research  &  Technology) 

SW-CMM 

60%  reduction  in  cost  of  customer  acceptance  (Thales  Research  &  Technology) 

SW-CMM 

Cost  variances  decreased  as  process  maturity  increased  (Thales  Training  and  Simulation) 

SW-CMM 

5  The  amount  of  detail  provided  in  the  published  reports  varies  considerably.  We  have  done  our  best 
to  supplement  these  reports  with  information  from  other  public  sources  and  our  previous  familiar¬ 
ity  with  the  organizations  involved. 
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4.1.2  Schedule 


This  category  covers  two  aspects  of  schedule:  improvements  in  schedule  predictability  and 
reductions  in  the  time  required  to  do  the  work.  Since  measures  of  productivity  are  generally 
based  on  the  amount  of  work  accomplished  in  a  period  of  time,  this  is  included  here  as  well. 
More  detail  about  each  of  these  instances  can  be  found  in  the  organizational  descriptions  in 
the  next  section. 


Table  2:  Summary  Benefits  and  Impact:  Schedule 


Result 

Model 

Reduced  by  half  the  amount  of  time  required  to  turn  around  releases  (Boeing,  Australia) 

CMMI 

60%  reduction  in  work  and  fewer  outstanding  actions  following  pre-test  and  post-test  audits 

(Boeing,  Australia) 

CMMI 

Increased  the  percentage  of  milestones  met  from  approximately  50%  to  approximately  95% 

(General  Motors) 

CMMI 

Decreased  the  average  number  of  days  late  from  approximately  50  to  fewer  than  10 

(General  Motors) 

CMMI 

Increased  through-put  resulting  in  more  releases  per  year  (JP  Morgan  Chase) 

CMMI 

30%  increase  in  software  productivity  (Lockheed  Martin  M&DS) 

CMMI 

Improved  and  stabilized  Schedule  Performance  Index  (Northrop  Grumman  IT1) 

CMMI 

Met  every  milestone  (25  in  a  row)  on  time,  with  high  quality  and  customer  satisfaction 

(Northrop  Grumman  IT2) 

CMMI 

10%  improvement  in  first  pass  yield  leading  to  reduction  in  rework  (Bosch  Gasoline  Systems) 

SW-CMM 

15%  improvement  in  internal  on-time  delivery  (Bosch  Gasoline  Systems) 

SW-CMM 

Improved  predictability  of  delivery  schedule  (JP  Morgan  Chase) 

SW-CMM 

Schedule  variances  decreased  as  process  maturity  increased  (Thales  Training  and  Simulation) 

SW-CMM 
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4.1.3  Quality 

Quality  improvement  is  most  frequently  measured  by  reductions  in  the  number  of  defects  at 
different  points  in  the  process  or  overall  in  the  product.  However,  we  recognize  that  there  are 
other  ways  to  characterize  and  measure  quality.  Where  these  are  found,  we  have  included 
them.  More  detail  about  each  of  these  instances  can  be  found  in  the  organizational  descrip¬ 
tions  in  the  next  section. 


Table  3:  Summary  Benefits  and  Impact:  Quality 


Result 

Model 

Met  goal  of  20  +/-  5  defects  per  KLOC  (Northrop  Grumman  IT1) 

CMMl 

Only  2%  of  all  defects  found  in  the  fielded  system  (Northrop  Grumman  IT1) 

CMMI 

Reduction  in  defects  found  from  6.6  per  KLOC  to  2. 1  over  5  causal  analysis  cycles 

(Northrop  Grumman  IT2) 

CMMI 

Increased  focus  on  quality  by  developers  (Northrop  Grumman  IT2) 

CMMI 

Reduction  in  error  cases  in  the  factory  by  one  order  of  magnitude  (Bosch  Gasoline  Systems) 

SW-CMM 

Reduction  in  number  and  severity  of  post-release  defects  (JP  Morgan  Chase) 

SW-CMM 

Most  of  $2  million  savings  resulted  from  early  detection  and  removal  of  defects 
(Sanchez  Computer  Associates,  Inc.) 

SW-CMM 

Improved  quality  of  code  (Sanchez  Computer  Associates,  Inc.) 

SW-CMM 

4.1.4  Customer  Satisfaction 

Few  organizations  have  reported  on  measures  of  customer  satisfaction;  however,  we  have 
included  this  section  because  of  its  importance  for  many  organizations  as  a  measure  of  the 
benefits  of  process  improvement.  More  detail  about  each  of  these  instances  can  be  found  in 
the  organizational  descriptions  in  the  next  section. 


Table  4:  Summary  Benefits  and  Impact:  Customer  Satisfaction 


Result 

Model 

Increased  award  fees  by  55%  compared  to  an  earlier  SW-CMM  baseline  at  maturity  level  2 

(Lockheed  Martin  M&DS) 

CMMI 

Received  more  than  98%  of  possible  customer  award  fees  (Northrop  Grumman  IT1) 

CMMI 

Earned  a  rating  of  “Exceptional”  in  every  applicable  category  on  their  Contractor  Performance 
Evaluation  Survey  (Northrop  Grumman  IT2) 

CMMI 

C  MU/SE 1-2003-S  R-009 


9 


4.1.5  Return  on  Investment  (ROI) 

ROI  can  be  calculated  using  a  variety  of  costs  and  benefits.  More  detail  about  each  of  these 
instances  can  be  found  in  the  organizational  descriptions  in  the  next  section. 


Table  5:  Summary  Benefits  and  Impact:  Return  on  Investment 


Result 

Model 

5:1  ROI  for  quality  activities  (Accenture) 

CMMI 

13:1  ROI  calculated  as  defects  avoided  per  hour  spent  in  training  and  defect  prevention 

(Northrop  Grumman  IT2) 

CMMI 

Processes  for  earlier  defect  detection,  improved  risk  management,  and  better  project  control  imple¬ 
mented  after  showing  positive  return  on  investment  during  pilot  (Thales  TT&S)6 

CMMI 

4.2  Organizations  Reporting  CMMI  Benefits 
4.2.1  Accenture 

Accenture  is  a  management  consulting  and  technology  services  organization  employing  more 
than  75,000  people  in  47  countries.  After  achieving  SW-CMM  maturity  level  3  in  some  parts 
of  the  organization,  Accenture  shifted  its  focus  to  CMMI  for  Systems  and  Software  Engineer¬ 
ing  (SE/SW)  with  integrated  product  and  process  development  (IPPD)  V  1.02.  The  improve¬ 
ment  effort  was  focused  on  six  large  projects  in  their  US  Government  Operating  Unit.  These 
groups  had  already  achieved  ISO  registration,  and  were  using  an  integrated  work  approach 
with  multi-disciplinary  teams  which  included  their  clients.  They  accomplished  their  transition 
to  the  CMMI  models  in  18  months,  from  May  2001  to  December  2002,  at  which  point  they 
held  a  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  V  1.1  ap¬ 
praisal  and  achieved  maturity  level  3. 

As  shown  in  Figure  2,  Accenture  expended  a  total  of  8,045  hours  on  its  transition  to  the 
CMMI  Product  Suite.  Sixty-four  percent  of  the  total  was  spent  on  process  development,  with 
the  rest  allocated  to  planning,  requirements,  and  design  of  new  processes  and  process  im¬ 
provements. 


6  TT&S  has  not  made  its  quantitative  results  on  return  on  investment  publicly  available. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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CMM'SWto  CM  Ml  Development  Hours  (approx) 

260  425 

■  Planning 

■  Requirements 
□  Design 

■  Development 


5  2002  Accenture  Ail  Rights  Reserved 

Figure  2:  Accenture  Transition  Hours 

In  its  CMMI  implementation,  Accenture  focused  on  their  processes  related  to  the  Decision 
Analysis  and  Resolution  process  area  (PA).  They  applied  those  practices  to  their  processes 
related  to  the  Technical  Solution,  Requirements  Management,  and  Project  Monitoring  and 
Control  PAs.  They  concentrated  on  the  model’s  generic  goals  as  well  as  the  Organizational 
Environment  for  Integration  PA  and  the  IPPD  concept  of  Shared  Vision.  The  unit  reported 
several  lessons  learned  from  this  experience.  By  focusing  early  on  Measurement  and  Analysis 
and  “thinking  level  4  and  5,”  they  gained  greater  management  capability.  Like  so  many  oth¬ 
ers,  they  reaffirmed  the  importance  of  securing  management  sponsorship,  providing  support 
to  projects  during  their  transition,  and  running  their  process  improvement  efforts  like  pro¬ 
jects.  Accenture  reported  that  it  was  possible  to  achieve  an  ROI  of  5:1  from  hours  invested  in 
quality  activities  [Bengzon  03]. 


4.2.2  Boeing  Australia,  Limited 

Boeing  Australia,  Limited,  a  wholly-owned  subsidiary  of  The  Boeing  Company,  is  a  high 
technology  aerospace  enterprise  that  develops  systems  for  both  defense  and  commercial  mar¬ 
kets.  With  their  head  office  in  Brisbane,  Queensland,  Boeing  employs  more  than  1,400  peo¬ 
ple  in  13  locations  around  the  country  [Boeing  03]. 

Between  1996  and  1999,  a  series  of  reviews  of  major  projects  that  develop  software-intensive 
systems  for  Australian  defense  organizations  raised  many  concerns  and  issues  about  process 
performance.  As  a  result,  the  Australian  Defence  Materiel  Organisation  (DMO)  developed 
objectives  for  reform  and  sponsored  a  broad  examination  of  the  potential  utility  of  capability 
maturity  models  and  methods  for  the  Australian  defense  industry.  They  determined  that 
improving  processes  based  on  a  continuous  representation  of  the  CMMI  models  would  best 
meet  their  needs  for  1)  coverage  of  both  systems  and  software  engineering,  2)  acceptance  by 
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suppliers  nationally  and  internationally,  and  3)  the  DMO  strategy  of  assessing  risk  against 
desired  capability  profiles. 

They  began  by  mapping  the  CMMI  Product  Suite  to  ISO/IEC  15504  [ISO  03],  and  partici¬ 
pated  in  several  early  pilots  of  the  SCAMPI  appraisal  methodology  [Marshall  et  al.  01].  In 
2001,  they  decided  to  continue  applying  both  the  CMMI  Product  Suite  and  15504  to  evaluate 
the  maturity  of  a  supplier’s  processes.* 7 

Boeing  Australia  has  a  heritage  of  improvement  based  both  on  EIA/  IS-731  [EIA  98]  and  the 
SW-CMM  before  making  the  transition  to  the  CMMI  Product  Suite.  As  part  of  a  broader 
presentation  that  reviews  many  general  topics  which  might  be  of  interest  to  organizations 
planning  to  implement  the  models,  Stevenson  cites  Boeing’s  submarine  project  as  an  example 
of  pertinent  CMMI  results.8  These  include  a  33%  decrease  in  the  average  cost  to  correct  a 
defect  over  an  18-month  period.  The  necessary  time  for  the  delivery  of  a  release  was  cut  in 
half,  with  an  increase  in  the  ability  to  configure  builds.  Stevenson  also  reports  a  60  percent 
reduction  in  the  preparation,  conduct,  and  rework  from  pre-test  and  post-test  audits,  resulting 
in  audits  passed  with  few  to  no  outstanding  actions.  More  qualitatively,  he  notes  that  Boeing 
Australia  developers  have  become  increasingly  focused  on  eliminating  defects  across  all 
products,  improving  quality,  and  finding  ways  to  improve  their  processes. 


4.2.3  General  Motors  Corporation 

General  Motors  Information  Systems  and  Services  was  formed  in  1996  to  provide  informa¬ 
tion  technology  management  and  technical  capability  within  the  corporation.  Process  im¬ 
provement  work  began  immediately.  From  1996  to  2000  they  focused  on  defining  the  system 
delivery  process  with  a  small,  central  group  that  attempted  to  create  a  “quantum  change” 
across  the  organization  using  a  shared  intervention  strategy.  In  2000  they  refocused  their 
leadership  effort  and  involved  more  practitioners  in  process  improvement.  They  approached 
improvement  using  a  more  balanced  and  gradual  transition  approach,  which  focused  on  or¬ 
ganizational  and  process  capabilities  and  the  impact  of  change  on  the  organization  [General 
Motors  03],  [Hofmann  et  al.  01]. 


Hofmann  et  al.  describe  their  improvement  effort  as  a  “CMMI-style  improvement  approach” 
[Hofmann  et  al.  03].  Using  an  appraisal  tool  to  “enable  consistent,  on-site  data  capture  and 
real-time  data  analysis,”  General  Motors  organizations  use  internal  appraisals  to  identify  gaps 
between  their  processes  and  the  CMMI  framework  of  best  practices.  Appraisal  results  are 


Some  of  the  information  is  provided  here  with  permission  from  a  proprietary  presentation  about 

Boeing’s  experience  using  capability  maturity  models. 

8  This  information  comes  from  a  presentation  by  Terry  Stevenson  titled  “Investing  in  Process — Is 
There  a  Return?”  The  presentation  was  made  at  the  Software  Engineering  Australia  2002  Confer¬ 
ence  in  Gold  Coast,  Queensland  Australia,  September  2-4, 2003. 
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reported  to  senior  management.  Improvement  plans  are  developed  and  executed  on  a  regular 
six-month  basis. 

The  organization  also  cites  the  early  implementation  of  a  measurement  program  as  a  key  to 
its  success.  By  starting  early,  they  gradually  increased  their  analytic  capability  as  they  gath¬ 
ered  more  improvement  data.  They  measured  organizational  awareness  and  process  compli¬ 
ance,  and  collected  product  measures.  Organizations  within  information  technology  (IT)  de¬ 
veloped  and  executed  six-month  improvement  plans,  which  were  driven  by  the  results  of 
causal  analyses  performed  on  their  measurement  data. 

As  noted  in  Figure  3,  General  Motors  organizations  met  their  schedules  more  consistently 
when  they  made  the  transition  to  CMMI  models.  The  number  of  project  milestones  met  in¬ 
creased  from  about  50  percent  to  about  85  percent.  Similarly,  they  reduced  the  average  num¬ 
ber  of  days  late,  from  more  than  50  to  fewer  than  10.  Notice  that  the  major  shift  in  both 
charts — most  noticeable  in  the  second— occurs  after  they  shifted  to  using  CMMI  models 
[Hofmann  et  al.  03]. 
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Figure  3:  General  Motors  Schedule  Data 

4.2.4  Lockheed  Martin  Management  &  Data  Systems 

Lockheed  Martin  Management  &  Data  Systems  (M&DS)  provides  systems  integration,  sys¬ 
tems  engineering,  software  development,  and  program  management  in  support  of  defense, 
civil,  international,  and  commercial  customers.  M&DS  is  a  $2.3  billion  unit  of  Lockheed 
Martin  Corporation  with  approximately  9,000  employees  at  facilities  in  Pennsylvania,  Ari¬ 
zona,  California,  Colorado,  and  the  Washington,  DC  area  [Lockheed  Martin  03]. 


M&DS  has  been  deeply  involved  in  process  improvement  for  more  than  ten  years.  They  were 
assessed  at  SW-CMM  Level  2  in  1993.  They  also  have  a  long  heritage  of  integrating  both 
software  and  systems  engineering.  M&DS  established  a  foundation  in  systems  engineering  in 
the  early  1990s.  In  2000,  M&DS  achieved  maturity  level  5  in  both  the  CMM  for  Systems 
Engineering  (SE-CMM)  and  in  the  SW-CMM  [Newswire  01].  After  18  months  of  additional 
effort,  they  were  appraised  at  CMMI  maturity  level  5  in  2002  [Ring  et  al.  02]. 
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Their  journey  from  SW-CMM  maturity  level  2  to  CMMI  maturity  level  5  (1995-2002)  has 
been  characterized  by  continuing  improvement.9 10  For  example,  they  have  documented  a  4.5 
percent  decrease  in  their  overhead  rate.  Moreover,  they  steadily  captured  a  greater  percentage 
of  available  award  fees  over  the  years,  now  receiving  55  percent  more  fees  compared  to  the 
baseline  that  remained  unrealized  at  SW-CMM  maturity  level  2. 


Concurrent  with  the  period  from  2000  to  2002  in  which  they  achieved  maturity  level  5 
against  both  the  SW-CMM  and  CMMI  models,  M&DS  implemented  a  number  of  design- 
related  engineering  methodologies,  tools,  and  process  initiatives.  They  also  focused  on  im¬ 
proving  the  quality  of  their  processes  for  defect  identification,  prevention,  and  removal.  No¬ 
table  performance  improvements  in  these  3  years  include  an  increase  in  software  productivity 
by  30  percent,  a  decrease  in  unit  software  cost  by  20  percent,  and  a  15  percent  decrease  in 
costs  to  find  and  fix  defects. 

4.2.5  Northrop  Grumman  Information  Technology! 

Northrop  Grumman  Information  Technology,  which  is  part  of  the  larger  Northrop  Grumman 
Corporation,  generates  about  $4  billion  sales  annually,  has  22,000  employees,  and  provides 
information,  engineering,  and  business  solutions  for  government,  commercial,  and  interna¬ 
tional  enterprises.  The  majority  of  their  work  is  in  software-intensive  systems.  Northrop 
Grumman  Information  Technology  maintains  an  ongoing  commitment  to  quality,  noting  on 
its  Web  site  that  the  organization  has  “made  continuing  process  improvement  one  of  our 
highest  priorities”  [Northrop  Grumman  03].  Their  Defense  Enterprise  Solutions  (DES)  busi¬ 
ness  unit  achieved  a  CMMI  maturity  level  5  rating  in  December  2002.'° 

DES  exemplifies  how  a  high  maturity  organization  can  undertake  continuous  improvement. 
Projects  define  goals  that  are  compatible  with  the  organization’s  defined  business  objectives. 
The  projects  then  track  and  measure  their  performance  using  common  measures  for  their  tai¬ 
lored  processes,  make  corrections,  and  feed  the  corrections  back  into  the  organization’s  set  of 
standard  processes.  For  example,  their  optimizing  strategy  for  defect  detection  and  prevention 
maps  nicely  with  the  Causal  Analysis  and  Resolution  process  area  in  the  CMMI  models.  De¬ 
fect  data  are  regularly  collected  and  analyzed,  looking  for  both  special  and  common  causes  of 


9  The  M&DS  process  improvement  journey  began  before  the  existence  of  the  CMMI  Product  Suite; 
however,  the  important  point  is  that  their  earlier  performance  improvements  are  attributable  to 
best  practices  that  have  since  been  incorporated  into  the  CMMI  models.  Additional  effort  was 
necessary  to  achieve  CMMI  maturity  level  5,  and  the  existence  of  the  CMMI  models  also  facili¬ 
tated  further  improvements  in  performance. 

10  DES  was  created  in  January  2002  by  joining  several  previously  separate  organizational  units  from 
Northrop  Grumman  and  its  recent  corporate  acquisitions.  Those  units  followed  separate  processes, 
and  they  ranged  in  maturity  level  from  SW-CMM  level  2  through  SE-CMM  and  SW-CMM  level 
5.  The  organization’s  commitment  to  quantitative  management  and  continuous  improvement  en¬ 
abled  them  to  combine  the  units  and  improve  their  overall  process  performance  in  a  very  short  pe¬ 
riod. 


14 


CMU/SEI-2003-SR-009 


variation.  Pareto  and  other  root  cause  analyses  are  performed,  actions  are  taken  to  remove 
root  causes  and  prevent  future  defects.  Changes  are  made  to  the  organization’s  defined  proc¬ 
ess  and  the  projects’  tailored  processes,  and  technological  innovations  are  introduced  where 
applicable.  Tests  or  peer  reviews  then  are  conducted  following  the  new  or  modified  proc¬ 
esses,  and  the  cycle  begins  again. 

Figure  4  displays  results  from  their  defect  density  review  process  in  one  large  project.  They 
set  a  goal  of  20  +/-  5  defects  per  thousands  lines  of  code  (KLOC).  Notice  on  the  left  side  of 
the  graph  that  they  found  significant  variation  in  defects  per  KLOC.  After  identifying  the 
special  causes  of  variation  and  implementing  an  improvement  in  the  process,  they  saw  sub¬ 
stantially  less  variation.  Only  two  percent  of  all  defects  were  found  in  the  fielded  system 
[Pflugrad  02]. 


Defect  Density  at  Review  (all  defects) 


Figure  4:  Northrop  Grumman  Defect  Density  Reviews 

Pflugrad  presents  similar  effects  for  the  same  project’s  cost  performance  index  (CPI)  and 
schedule  performance  index  (SPI)  as  a  result  of  improvements  in  processes  for  estimation  and 
earned  value  tracking.  In  the  area  of  customer  satisfaction,  they  are  receiving  more  than  98 
percent  of  the  possible  award  fees  from  their  customer,  compared  to  a  goal  of  equal  to  or 
greater  than  95  percent  [Pflugrad  02]. 

As  shown  in  Figure  5,  a  decline  in  defect  density  on  another  project  was  accompanied  by  an 
increase  in  customer  satisfaction.  Interestingly,  the  defect  density  comparison  was  added  in 
the  later  time  period,  as  they  moved  toward  maturity  level  5. 


CMU/SEI-2003-SR-009 


15 


Figure  5:  Northrop  Grumman  Defect  Density  and  Customer  Satisfaction 


4.2.6  Northrop  Grumman  Information  Technology2 

Northrop  Grumman  used  the  Personal  Software  ProcessSM  (PSP514)  in  conjunction  with 
CMMI  maturity  level  5  practices  on  another  project  that  built  an  Inventory  Tracking  System. 
The  developers  used  the  PSP  to  collect  individual  data.  These  data  were  aggregated  to  facili¬ 
tate  the  measurement  of  time  in  phase,  defect  density  by  phase,  review  rates  and  effective¬ 
ness,  and  schedule  performance.  They  also  measured  requirements  volatility,  product  size, 
earned  value,  defect  density,  and  customer  satisfaction  as  part  of  their  implementation  of 
Quantitative  Project  Management  at  maturity  level  4.  Finally,  through  their  maturity  level  5 
optimizing  process  that  implements  the  Causal  Analysis  and  Resolution  PA,  they  performed 
causal  analyses  on  defects  for  each  build  cycle.  At  that  time,  they  captured  data  by  unit  and 
build  on  defect  injection,  removal,  severity,  defect  type,  and  fix  time. 


Figure  6  shows  results  over  five  builds  of  the  tracking  system.  Northrop  Grumman  was  able 
to  identify  particularly  important  defect  types  after  the  first  three  defect  prevention  cycles. 
Following  the  first  three  builds,  they  undertook  a  number  of  countermeasures  (e.g.,  revising  a 
design  document  template,  revising  peer  review  checklists,  emphasizing  the  need  for  ade¬ 
quate  review  time,  and  performing  “global”  reviews  on  certain  types  of  defects).  As  seen  in 
Figure  6,  defect  density  found  in  the  peer  reviews  decreased  markedly  after  the  second  cycle. 
After  a  noticeable  upturn  on  the  fourth  build,  defect  density  again  decreased  considerably  on 
the  fifth  build.  Hoffman  concludes  that  “[causal  analysis]  cycles  can  provide  a  tremendous 
cost  savings  for  a  very  modest  effort,  quite  rapidly.”  Indeed,  he  reports  a  return  on  investment 
of  13:1,  calculated  as  defects  avoided  per  hour  spent  in  training  and  defect  prevention.* 11 


SM  Personal  Software  Process  and  PSP  are  service  marks  of  Carnegie  Mellon  University. 

11  See  Hoffman,  G,  “Integrating  PSP5111  and  CMMI®  Level  5,”  May  2003.  This  slide  presentation 
may  be  found  in  the  Software  Engineering  Information  Repository  at  http://seir.sei.cmu.edu. 
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Figure  6:  Northrop  Grumman  Build  Cycles 
Other  benefits  that  Hoffman  attributes  to  their  process  improvement  effort  include 

•  meeting  every  milestone  (25  in  a  row)  on  time,  with  high  quality  and  complete  customer 
satisfaction 

•  an  increased  focus  by  developers  on  product  quality 

•  earning  a  rating  of  “Exceptional”  in  every  applicable  category  on  their  Contractor 
Performance  Evaluation  Survey 

•  finding  that  their  technical  staff  are  providing  superior  estimates  and  demonstrate  en¬ 
hanced  understanding  because  they  see  their  data  being  used  immediately  and  are  in¬ 
volved  in  a  full  plan-do-check-act  cycle 

4.2.7  Thales  Air  Traffic  Management  (ATM) 

The  Thales  Group,  a  global  electronics  company  with  65,000  employees  in  more  than  30 
countries,  generated  11.1  billion  euros  in  revenues  in  2002.  They  have  three  major  business 
units:  Aerospace,  Defence,  and  Information  Technology.  Air  Traffic  Management  (ATM)  is 
one  of  three  major  units  in  the  Aerospace  zone  [Thales  ATM  03a].  ‘Thales  ATM’s  mission  is 
to  support  the  Air  Transportation  Community  on  a  worldwide  basis  by  building  safe,  efficient 
and  inter-connectable  Communication  Navigation  Surveillance/ Air  Traffic  Management 
(CNS/ATM)  systems.  Our  solutions  are  based  on  consistent  global  development  processes, 

•  programme  management  and  systems  engineering,  combined  with  a  predictable  quality  base¬ 
line....  Quality  is  at  the  heart  of  what  we  do.  We  are  dedicated  to  continual  improvement  of 
our  own  processes....”  [Thales  ATM  03b]. 

Thales  ATM  has  progressed  from  SW-CMM  maturity  level  3  to  CMMI  maturity  level  4. 
Goeyse  et  al.  describe  how  their  high  maturity  CMMI  practices  have  supported  meeting  their 
business  objectives  [Goeyse  et  al.  03].  Measurement  practices  are  supported  by  training  and 
an  organizational  metrics  dictionary.  They  have  identified  decisions  requiring  Decision 
Analysis  and  Resolution  techniques  within  their  process  diagrams.  Risks  are  tracked,  moni- 
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tored,  stored  in  a  database,  and  actions  are  taken  where  necessary.  They  have  brought  an  in¬ 
creasing  number  of  processes  under  statistical  process  control,  for  example,  production  and 
verification  of  requirements,  design,  and  code  processes.  They  also  use  measurement  results 
to  direct  their  process  improvement  efforts.  Program,  company,  and  team  objectives  are  es¬ 
tablished  based  on  process  performance  and  measured  by  company  metrics.  Objectives  are 
set  by  business  needs  and  customer  negotiation  and  addressed  through  improvement  initia¬ 
tives.  Each  improvement  is  piloted  and  ROI  is  measured  before  it  is  rolled  out  to  the  organi¬ 
zation.  Goeyse  et  al.  highlight  earlier  defect  detection,  improved  risk  management,  and  better 
project  control  as  major  benefits  of  the  Thales  ATM  process  improvement  effort,  although 
they  do  not  provide  specific  quantitative  results  [Goeyse  et  al.  03]. 


4.2.8  Thales  Training  &  Simulation  (TT&S) 

Thales  TT&S  is  part  of  the  aerospace  business  unit,  which  is  one  of  three  major  areas  in  this 
international  company.  With  about  2,000  staff  in  Australia,  France,  the  United  Kingdom,  and 
the  United  States,  TT&S  develops  simulation,  systems,  and  training  services  for  both  military 
and  civil  applications  [Richard  03a],  [Richard  03b]. 

TT&S  initiated  an  effort  in  1992  to  improve  their  software  processes  based  on  SW-CMM. 
They  achieved  maturity  level  2  in  March  of  1995,  and  level  3  in  October  1996.  After  explor¬ 
ing  the  use  of  the  SE-CMM  to  better  manage  their  systems  engineering  processes,  they 
turned  their  focus  to  CMMI V  1.0. 

Their  goal  was  to  achieve  CMMI  maturity  level  2  for  project  management,  systems  engineer¬ 
ing,  quality,  purchasing,  and  the  hardware  and  software  interfaces  with  systems  engineering. 
To  that  end,  they  developed  and  implemented  process  and  organizational  changes,  using  the 
Measurement  and  Analysis  PA  to  link  their  process  changes  with  their  business  objectives. 
Specifically,  they  identified  financial  and  quality  objectives  which  were  cascaded  down 
through  the  organization  to  the  business  units  and  departments,  and  they  put  tools  and  prac¬ 
tices  into  place  to  measure  achievement  of  these  objectives.  New  processes  were  introduced 
to  the  organization  through  training,  coaching,  management  meetings  and  other  tools  and 
methods  of  deployment.  They  conducted  over  twelve  internal  assessments  in  eight  areas,  be¬ 
ginning  in  March  2000.  They  achieved  CMMI  maturity  level  2  in  one  of  the  earliest  SCAMPI 
appraisals  in  September  2001. 

Thales  TT&S  reports  a  number  of  lessons  learned  [Richard  03a],  [Richard  03b].  First,  they 
found  that  conducting  internal  “CBA IPI  like”  assessments  on  a  quarterly  basis  has  provided 
progress  measures  which  allow  them  to  react  quickly  if  adverse  trends  are  identified.  This 
practice  was  initiated  as  a  way  to  avoid  regression  after  successful  implementation  of  CMM- 
based  changes.  It  also  helps  them  with  deployment  of  new  processes  and  to  gather  data  to 
evaluate  achievement  of  their  objectives.  They  view  the  experience  gained  during  their  im¬ 
plementation  of  the  SE-CMM  and  SW-CMM  as  a  key  factor  in  their  success  with  the  CMMI 
Product  Suite. 


18 


CMU/SEI-2003-SR-009 


Richard  presents  data  on  funds  invested  in  the  organization’s  SEPG  to  provide  training,  con¬ 
duct  internal  appraisals,  document  their  processes  using  an  intranet  application,  and  support 
their  SCAMPI  appraisal.  He  states  that  analysis  of  data  collected  on  software  has  shown  that 
project  cost  and  schedule  variances  decreased  as  TT&S  process  maturity  increased  [Richard 
03a],  [Richard  03b). 


4.3  Organizations  Reporting  Benefits  from  SW-CMM 

The  following  vignettes  show  recent  evidence  of  impact  and  benefits  attributed  to  process 
improvement  based  on  the  Software  CMM.  We  include  them  here  since  the  SW-CMM  is  one 
of  the  source  models  for  CMMI.  Moreover,  all  of  these  organizations  are  moving  to  CMMI 
models. 

4.3.1  Bosch  Gasoline  Systems 

The  Bosch  Group  is  comprised  of  three  business  sectors:  Automotive  Technology,  Industrial 
Technology  and  Consumer  Goods,  and  Building  Technology.  The  Automotive  Technology 
Sector,  which  includes  Bosch  Gasoline  Systems,  generates  two-thirds  of  the  group’s  sales, 
which  reached  23.3  billion  euros  in  2002. 

The  Gasoline  Systems  Division  has  19,022  employees,  1000  of  whom  work  on  engine  con¬ 
trol  units,  software  development,  and  calibration.  They  recognized  that  software  had  become 
an  increasingly  critical  part  of  their  product  and  processes,  and  that  it  was  requiring  a  grow¬ 
ing  percentage  of  their  development  time  and  costs  [Bosch  03]. 

Process  improvements  had  been  attempted  prior  to  2001,  but  with  little  success  [Stolz  et  al. 
03].  The  Bosch  Gasoline  Systems  management  board  then  included  process  improvement 
and  adoption  of  the  SW-CMM  reference  model  in  their  business  goals.  The  current  software 
process  improvement  initiative  started  in  April  2001.  The  organization  is  a  large,  matrixed 
structure.  Stolz  et  al.  describe  their  senior  management  steering  group,  involvement  of  both 
line  managers  and  experienced  staff,  and  the  use  of  measurement  to  conduct  audits  and  inter¬ 
nal  assessments  as  critical  success  factors  [Stolz  et  al.  03].  Their  measurement  “cockpit”  re¬ 
port  displays  graphs  for  17  projects  and  6  processes:  Project  Planning,  Project  Tracking  and 
Oversight,  Software  Quality  Assurance,  Requirements  Management,  Configuration  Manage¬ 
ment,  and  Subcontract  Management.  Their  results  overall  show  a  positive  trend. 

Stolz  et  al.  summarize  qualitatively  several  benefits  of  having  achieved  SW-CMM  level  3: 

•  clearly  defined  project  requirements 

•  improved  predictability  of  their  work  packages 

•  clearly  defined  interfaces  for  change  requests  and  requirements  management 
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•  improved  ability  to  manage  complex  systems  and  projects 

•  better  project  and  risk  management  [Stolz  et  al.  03] 

Quantitative  improvements  include 

•  a  15  percent  improvement  in  internal  on-time  delivery 

•  a  10  percent  improvement  in  first-pass  yield  leading  to  reduction  in  rework 

•  a  substantial  (order  of  magnitude)  reduction  in  error  cases  over  2  years  [Stolz  et  al.  03] 

Next  steps  include  1)  moving  to  the  CMMI  Product  Suite  and  applying  it  to  software,  sys¬ 
tems,  and  hardware  development;  2)  expanding  their  process  improvement  program  to  in¬ 
clude  sales,  hardware,  and  component  development;  and  3)  tracking  their  progress  against 
CMMI  maturity  levels. 


4.3.2  JP  Morgan  Chase  &  Company,  Investment  Bank 
Technology 

Investment  Bank  Technology,  a  unit  of  JP  Morgan  Chase  and  Company,  is  a  global  IT  organi¬ 
zation  supporting  a  global  financial  services  firm.  They  provide  technology  and  applications 
development  support  for  business  lines  in  nine  countries.  Quality  systems  and  software  appli¬ 
cations  are  increasingly  critical  to  financial  institutions,  as  is  rapid  development  of  new  tech¬ 
nology.  The  financial  impact  of  a  software  defect  can  be  enormous. 

The  Investment  Bank  Technology  unit  in  the  United  Kingdom  began  to  implement  process 
improvement  based  on  the  SW-CMM  with  one  pilot  project  in  2001;  they  now  have  28  teams 
that  have  been  appraised  at  SW-CMM  maturity  level  2.  The  first  team  to  successfully  im¬ 
prove  using  the  CMMI  Product  Suite  achieved  a  maturity  level  3  rating  in  2003.  They  are 
now  using  CMMI  models  along  with  several  other  improvement  techniques,  including  Six 
Sigma,  across  the  oiganization. 

The  unit  reports  several  benefits  in  quality  and  time  from  these  process  improvements,  in¬ 
cluding 

J 

•  reduction  in  and  reduced  severity  of  post-release  defects 

•  improved  predictability  of  delivery  schedule 

Notably,  their  CMMI  maturity  level  3  team  has  reported  a  major  increase  in  through-put  re¬ 
sulting  in  more  releases  per  year.  Ongoing  analyses  are  showing  a  substantial  return  on  in¬ 
vestment.12 


12  This  information  is  taken  with  permission  from  a  proprietary  presentation  delivered  at  the  SEI. 
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4.3.3  Sanchez  Computer  Associates,  Inc. 

Sanchez  Computer  Associates,  with  annual  revenues  of  $93  million  in  2002,  provides  inte¬ 
grated  software  and  services  for  banking,  securities,  customer  integration,  wealth  manage¬ 
ment  and  outsourcing  to  nearly  400  financial  institutions  in  21  countries  [Sanchez  03a].  They 
set  their  sights  on  SW-CMM  maturity  level  3,  using  what  they  called  the  “Big  Bang”  ap¬ 
proach.  This  consisted  of  documenting  current  processes,  identifying  and  filling  gaps,  then 
training  and  rolling  out  the  changes.  They  were  appraised  at  SW-CMM  maturity  levels  2  and 
3  in  just  more  than  15  months.  They  report  the  following  benefits: 

•  a  robust  training  program 

•  the  application  of  process  to  activities  other  than  programming,  such  as  documentation 

•  improved  code  quality 

•  a  savings  of  $2  million  in  the  first  6  months,  primarily  from  early  detection  and  removal 
of  defects 

Sanchez  Computer  Associates  state  in  a  news  release  that:  “Achieving  a  CMM  Level  3  rating 
sets  us  apart  from  our  competitors  and  identifies  us  as  an  organization  committed  to  creating 
institutionalized  processes  and  procedures,  which,  in  turn,  allow  us  to  deliver  consistent  qual¬ 
ity  to  our  customers”  [David  Consulting  03],  [Sanchez  03]. 


4.3.4  Thales  Research  &  Technology,  UK 

As  part  of  a  general  tutorial  on  process  improvement.  Marsh  and  Vigier  present  an  overview 
of  the  techniques  and  methods  used  at  Thales  Research  &  Technology  in  the  United  Kingdom 
for  their  CMMI  based  process  improvement  effort  [Marsh  &  Vigier  03].  In  the  presentation, 
they  use  an  earlier  SW-CMM  example  from  another  Thales  unit  to  demonstrate  the  value  of 
process  improvement.  The  results  show  marked  improvements  in  cost  performance  as  the 
unit  progressed  from  SW-CMM  maturity  level  1  to  maturity  level  3.  Figure  7  shows  a  classic 
reduction  in  both  the  average  and  variability  of  cost  deviation.  In  addition,  the  sidebar  shows 
a  60  percent  reduction  in  cost  of  customer  acceptance,  due  to  reductions  in  defects  found  at 
customer  acceptance  and  resultant  late  acceptance. 
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10  Thales  Research  &  Technology  UK  THALES 


Figure  7:  Thales  Research  &  Technology,  UK 


What  is  of  particular  interest  here  is  the  fact  that  SW-CMM  results  are  used  to  forecast  the 
benefits  of  process  improvement  using  CMMI  models. 
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5  Current  Work  and  Plans  for  the  Future 


In  this  section,  we  describe  the  ongoing  effort  at  the  SEI  and  elsewhere  to  document  the  im¬ 
pact  and  benefits  of  process  improvement  based  on  CMMI  models.  The  section  begins  with  a 
description  of  an  ongoing  series  of  in-depth  case  studies.  That  is  followed  by  a  short  sum¬ 
mary  of  our  literature  review.  Next  is  a  brief  discussion  of  the  limitations  of  case  study  re¬ 
sults,  such  as  those  presented  in  this  report  as  well  as  the  ongoing  series  of  case  studies. 
While  case  studies  provide  a  great  deal  of  valuable  detail  and  context,  their  results  cannot 
necessarily  be  generalized  elsewhere.  Finally,  we  outline  our  plans  to  address  the  limitations 
of  using  case  studies  alone  along  with  our  plans  for  the  future. 


5.1  In-Depth  Case  Studies 

In  addition  to  this  special  report,  the  SEI  has  initiated  a  broader  research  effort  to  provide 
credible,  objective  evidence  about  the  value  of  CMMI-based  process  improvement.  As  a  first 
step,  we  have  entered  into  collaborative  relationships  with  several  early  adopters  to  conduct 
in-depth  case  studies  of  their  experiences  with  process  improvement  based  on  CMMI  models. 
Our  emphasis  is  on  quantitative  results  about  the  impact  and  value  added  by  CMMI.  The 
quantitative  analyses  are  being  augmented  by  qualitative  descriptions  of  the  participating  or¬ 
ganizations  and  their  process  improvement  efforts. 

Some  of  the  results  presented  in  this  report  have  been  normalized  to  protect  proprietary  in¬ 
formation.  The  same  will  be  done  in  future  analyses.  SEI  staff  members  are  reviewing  the 
data  and  results  to  better  understand  them  and  confirm  their  accuracy.  We  at  the  SEI  also  are 
providing  assistance  with  study  design  and  analysis  as  appropriate.  In  addition,  participating 
organizations  are  asked  to  provide  their  evidence  and  contextual  information  following  a 
standard  format  to  facilitate  summarization  and  quality  assurance.  For  example,  they  are 
asked  to  discuss  their  measurement  criteria,  including  evidence  that  the  results  can  properly 
be  attributed  to  CMMI-based  processes  as  opposed  to  other  factors  or  unintended  measure¬ 
ment  effects. 

As  in  this  report,  evidence  to  be  presented  will  include  a  variety  of  performance  dimensions. 
In  addition  to  return  on  investment  and  related  cost-benefit  measures,  these  include  cost,  pro¬ 
ductivity,  cycle  time,  predictability  of  cost  and  schedule,  and  product  quality.  Where  avail¬ 
able,  additional  evidence  will  be  presented  about  the  conditions  of  successful  adoption,  tran- 


CMU/SEI-2003-SR-009 


23 


sition,  and  documented  improvement;  these  may  include  discussion  of  the  pitfalls  and  obsta¬ 
cles  to  successful  adoption  and  use. 


Results  from  the  first  round  of  these  case  studies  are  included  in  a  track  on  CMMI  impact  and 
benefits  at  the  3rd  CMMI  Technology  Conference  and  User  Group  in  November  2003.  The 
track  also  includes  five  presentations  that  were  submitted  to  the  conference  without  our  prior 
knowledge.  We  expect  to  include  them  and  others  in  an  ongoing  series  of  such  studies.  The 
conference  track  also  includes  a  presentation  summarizing  the  first  15  cases  along  with  sup¬ 
plementary  evidence  provided  by  others.  In  addition,  the  track  includes  a  roundtable  panel  of 
experts  who  discuss  what  is  necessary  to  better  make  the  case  about  the  benefits  of  CMMI- 
based  process  improvement.  A  list  of  the  track  participants  may  be  found  in  the  Appendix. 


5.2  Literature  Review 

As  part  of  our  ongoing  research,  we  are  conducting  a  review  of  recent  literature.  Sources  of 
information  reviewed  thus  far  include  the  Software  Engineering  Process  Group  (SEPG), 
European  SEPG  and  Software  Technology  Conference  (STC)  conference  proceedings  from 

2000  through  2003,  and  the  CMMI  Technology  Conference  and  User  Group  proceedings  for 

2001  and  2002.  We  also  have  conducted  several  systematic  searches  through  journal  refer¬ 
ences  and  related  publications.  We  have  looked  first  for  recent  quantitative  results,  although 
we  also  have  flagged  publications  with  compelling  qualitative  results.  Thus  far,  we  have 
found  few  published  results  with  quantitative  evidence  about  the  impact  and  benefits  of 
CMMI  models.  The  same  is  true  for  recent  reports  based  on  the  SW-CMM. 


5.3  Generalizability 

The  results  presented  in  Section  4  provide  useful  early  evidence  about  the  benefits  of  CMMI- 
based  process  improvement.  But  what  does  it  take  to  enhance  the  quantity  and  quality  of  such 
evidence?  Additional  objective  evidence  also  is  needed  about  the  conditions  of  successful 
adoption  and  transition  to  CMMI-based  process  improvement. 

These  twelve  case  studies  are  confined  to  selected  reports  from  early  adopters  of  CMMI 
models.  In  that  sense,  their  results  are  existence  proofs  of  what  can  happen  under  the  right 
organizational  and  technical  circumstances.  Many  of  them  are  exemplary  organizations  that 
have  chosen  to  share  their  experiences  publicly.  Over  time  we  will  need  to  include  a  more 
broadly  based  sample.  Our  field  also  needs  to  pay  more  attention  to  issues  of  measurement, 
validation,  data  quality,  and  analytic  methods. 

Along  with  the  cases  presented  in  this  report,  the  organizations  participating  in  the  other,  in- 
depth  case  studies  described  in  Section  5.1  are  chosen  on  the  basis  of  their  access  to  existing 
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quantitative  evidence  and/or  their  willingness  to  collaborate  proactively  on  future  studies. 
However,  they  by  no  means  can  be  construed  as  a  comprehensive  sample  of  all  organizations 
that  have  begun  to  implement  process  improvement  programs  based  on  CMMI  models. 
CMMI V  1.1  was  released  in  January  2002.  It  still  is  fairly  new  and  many  organizations  are 
just  starting  to  apply  it.  Only  a  relative  few  of  them  have  come  forward  to  share  evidence  of 
quantitative  results  from  their  initial  improvement  efforts. 


5.4  Next  Steps 

More  credible  quantitative  information  about  CMMI  impact  exists  than  has  been  made  avail¬ 
able  publicly  thus  far.  As  of  June  2003,  for  example,  100  SCAMPI  VI.  1  appraisals  had  been 
conducted  and  reported  to  the  SEI.  Fully  one-fourth  of  them  were  appraised  at  high  maturity 
levels,  4  of  them  at  level  4  and  21  at  level  5.  Of  course,  all  maturity  level  5  and  many  level  4 
organizations  ought  to  have  some  quantitative  evidence  about  the  impact  of  process  im¬ 
provement.  A  growing  number  of  less  mature  organizations  also  now  appear  to  have  such 
evidence.13  Our  task  is  to  design  future  studies  that  better  reflect  the  experiences  of  the  wider 
CMMI  community. 

The  SEI  will  continue  to  work  collaboratively  with  organizations  that  are  willing  to  provide 
credible  objective  results  about  their  experiences  using  CMMI  models.  Relatively  little  quan¬ 
titative  evidence  currently  exists  publicly  about  the  impact  and  benefits  of  systems  engineer¬ 
ing  processes.14  We  will  pay  particular  attention  in  future  in-depth  case  studies  to  document¬ 
ing  the  results  of  disciplined  engineering  processes  in  largely  software  organizations.  We  also 
will  address  CMMI  impact  in  other  disciplines  in  addition  to  software  and  systems  engineer¬ 
ing. 

Case  studies  can  provide  much  useful  detail  that  helps  explain  their  results  in  proper  context, 
but  there  is  always  a  trade-off  between  breadth  and  depth.  As  just  mentioned  in  Section  5.3, 
results  from  selected  case  studies  are  not  necessarily  generalizable  to  a  wider  population.  In 
addition  to  our  collaborative  case  studies,  we  will  solicit  self-reported  cases  via  the  Software 
Engineering  Information  Repository  (SEIR)  [SEI  03].  As  part  of  that  effort,  and  in  addition  to 
qualitative  lessons  learned,  we  will  begin  to  collect  and  benchmark  standard  measures  of  ef¬ 
fort  and  performance  over  a  larger  group  of  organizations  than  we  can  possibly  work  with 


13  See,  for  example,  the  organizations  in  the  Appendix  who  report  on  doing  Causal  Analysis  and 
Resolution  related  activities  at  lower  levels  of  organizational  maturity.  The  existence  of  the 
Measurement  and  Analysis  process  area  at  maturity  level  2  in  CMMI  models  also  appears  to  be 
encouraging  earlier  attention  to  gathering  and  using  such  evidence  to  help  inform  decision  mak¬ 
ing. 

14  Organizations  that  have  used  the  SW-CMM  sometimes  have  covered  systems  engineering  proc¬ 
esses  in  their  improvement  efforts;  however,  quantitative  results  about  the  engineering  processes 
per  se  are  not  widely  available. 
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individually.  In  a  related  vein,  we  will  conduct  surveys  and  related  studies  of  CMMI  impact 
and  factors  that  affect  its  success. 

Other  emphases  for  fiscal  year  2004  and  beyond  may  include 

•  CMMI  adoption  and  impact  in  small  and  medium  enterprises 

•  research  and  development  on  the  costs  and  benefits  of  CMMI  appraisal  methods 

•  guidance  on  calculating  ROI  and  related  cost-benefit  information 
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6  Summary  and  Conclusion 


The  results  presented  in  this  report  provide  credible  early  evidence  that  CMMI-based  process 
improvement  can  result  in  better  project  performance  and  higher  quality  products.  The  results 
are  drawn  from  organizations  in  Europe  and  Australia  as  well  as  the  United  States.  Their 
process  improvement  efforts  cover  both  small  and  large  organizational  units.  They  do  busi¬ 
ness  in  a  variety  of  sectors  and  domains,  and  they  apply  CMMI  model  practices  to  systems 
integration,  systems  engineering,  and  software  development. 

As  noted  in  Section  3,  many  possible  measures  of  performance  can  be  used  to  demonstrate 
the  impact  of  CMMI-based  process  improvement.  We  have  gathered  here  results  that 
organizations  have  shared  with  us  or  with  the  wider  community  in  public  forums.  As  shown 
in  Figure  1,  we  have  categorized  the  results  into  four  primary  classes  of  benefits:  cost, 
schedule,  quality,  and  customer  satisfaction.  Evidence  about  return  on  investment  and  related 
cost-benefit  matters  constitutes  our  fifth  performance  category.  The  following  list  summa¬ 
rizes  the  results  from  the  twelve  cases  for  each  of  the  five  classes  of  performance  measures: 

•  Cost:  Six  cases  provide  nine  examples  of  cost-related  benefits,  including  reductions  in 
the  cost  to  find  and  fix  a  defect,  and  overall  cost  savings. 

•  Schedule:  Eight  cases  provide  evidence  of  schedule-related  benefits,  including  decreased 
time  needed  to  complete  tasks  and  increased  predictability  in  meeting  schedules. 

•  Quality:  Five  cases  provide  evidence  of  measurable  improvements  in  quality,  mostly 
related  to  reduction  of  defects  over  time  or  by  product  life  cycle. 

•  Customer  Satisfaction:  Three  cases  show  improvements  in  customer  satisfaction,  in¬ 
cluding  demonstration  of  customer  satisfaction  through  award  fees. 

•  Return  on  Investment:  Three  cases  claim  a  positive  return  on  investment  from  their 
CMMI-based  process  improvement,  and  two  of  these  provided  the  actual  results  of  then- 
calculations. 

The  twelve  cases  presented  here  are  based  on  selected  reports  from  early  adopters  of  the 
CMMI  Product  Suite.  They  are  exemplary  organizations  that  have  chosen  to  share  their  ex¬ 
periences  publicly;  however,  they  do  not  constitute  a  sample  of  all  organizations  that  are  or 
will  be  using  the  CMMI  models  to  guide  their  process  improvement  efforts.  Their  results  are 
evidence  of  what  can  happen  under  the  right  circumstances;  however,  additional  evidence 
also  is  needed  about  the  factors  that  may  help  explain  failure  as  well  as  success. 
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There  is  quite  a  bit  of  variety  among  these  results;  however,  as  Herbsleb  et  al.  note  in  their 
widely  read  1994  report,  the  results  also  come  from  a  variety  of  organizations  [Herbsleb  et  al. 
94].  The  diversity  of  the  organizations  is  echoed  in  the  diversity  of  the  information  they 
gather  and  report  as  evidence  that  their  hard  work  at  process  improvement  is  paying  off.  We 
hope  that  this  report  will  prove  to  be  useful  for  readers  who  wish  to  use  the  CMMI  Product 
Suite  to  enhance  their  own  process  improvement  efforts.  Our  hope  also  is  that  this  and  subse¬ 
quent  studies  will  inform  the  further  development  and  evolution  of  the  CMMI  Product  Suite. 
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Appendix  3rd  Annual  CMMI  Technology 

Conference  and  User  Group: 
Track  on  Impact  and  Benefits 
of  CMMI 


Author(s) 

Organization(s) 

Title 

Session  1:  Presentations 

David  Struble 

Raytheon 

Automating  Support  for  CMMI  Level  5  Organ¬ 
izational  Improvement 

Donna  Freed 

Raytheon 

CMMI  Process  Deployment  for  Software 

Doug  Brindley, 

Software  Productivity  Research 

Measurable  &  Predictable  Model  Based  Im¬ 

KV  Seshadri  Iyer 

MitoKen  Solutions 

provement 

Session  2:  Presentations 

Sarah  Bengzon 

Accenture 

Moving  On  to  CMMI  (SW/SE/IPPD)  Level  4 

Gary  Natwick 

Harris  Corporation 

Understanding  the  CMMI  Validation  PA 

Rick  Hefner, 

Dean  Caccavo 

Northrop  Grumman  Mission  Systems 

CMMI  Benefits  at  Northrop  Grumman  Mission 

Systems 

Session  3:  Presentations 

Ralph  Williams 

Cooliemon,  LLC 

Establishing  and  sustaining  a  Successful  Proc¬ 
ess  Group  (PG) 

Rolf  Reitzig 

cognence,  inc. 

Understanding  and  Improving  Software  Engi¬ 
neering  Economics 

Joan  Weszka 

Lockheed  Martin 

Transition  from  SW-CMM  to  CMMI:  The 

Benefits  Continue! 

Session  4:  Presentations 

Pete  McLoone 

Lockheed  Martin  Management  &  Data 

Systems 

Key  Business  Indicator  Trends  During  the 

Journey  from  SW-CMM  Level  2  to  CMMI 

Level  5  at  Lockheed  Martin  Management  & 

Data  Systems 
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Craig  Hollenbach 

Northrop  Grumman  IT 

Quantitatively  Measured  Process  Improvements 

at  Northrop  Grumman  IT 

Larry  McCarthy 

Motorola 

CMMI  Transition  Experience  and  Results  at 

Motorola 

Session  5:  Presentations 

Ralph  Williams 

Cooliemon,  LLC 

CAR  at  Maturity  Level  1 

Ruth  Buys 

CMS  Information  Services,  Inc. 

Evidence  of  CMMI  Impact 

Dennis  Goldenson, 

Diane  Gibson 

Software  Engineering  Institute 

Why  Should  I  Switch  to  CMMI?  Initial  Evi¬ 
dence  about  Impact  and  Value  Added 

Session  6:  Round  Robin  Panel 

Terry  Rout, 

Joe  Jarzombek, 

Khaled  El-Emam, 

Bob  Stoddard, 

Scott  Lucero 

Software  Quality  Institute 

OASD  (Nil)  IA 

National  Research  Council,  Canada 

Motorola 

OUSD  (AT&L/  Defense  Systems) 

Evidence  about  the  Benefits  of  CMMI:  What 

We  Already  Know  and  What  We  Need  to 

Know 
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